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Abstract 

We report on a search for the exclusive two-body charmless hadronic B meson decays B ?]' p, 
B —> rj'K*, B^ r]'(j), B^ rj'cv and B^ ri'ri^'\ The results are obtained from a data sample 
containing 535 xlO^ BB pairs that were collected at the T(4S') resonance with the Belle detector 
at the KEKB asymmetric-energy e~^e~ collider. We find no significant signals and report upper 
limits in the range (0.5-6.5) x 10"^ for all of the above decays. 

PACS numbers: 13.25.Hw, 11.30.Er 
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INTRODUCTION 



Information on the two-body charmless hadronic B meson decays with an rj' meson in 
the final state {B rj'h^*^) is incomplete at present. While the decay B — > rj'K is ob- 
served with a large branching fraction, so far no other B rj'h^*^ decay mode has been 
observed with greater than So" significance. The first evidence of i? ^ ri'ir has recently 
been reported [l|, HI and BaBar found evidence for B rj'K* with larger than signif- 
icance j3|, and thus additional observations are expected in the near future. The study of 
these decay modes can improve the understanding of the fiavor-singlet penguin amplitude 
with intermediate t, c and u quarks Furthermore, these studies increase our confidence 
in the reliability of a variety of other predictions, e.g, for the CP violating parameter 03 (V), 
and are necessary to extract theory parameters such as the scalar penguin operator |j, |5| . 
Presently, theoretical predictions for the branching fractions of these decay modes cover the 
range (0.0001-7.6) X 10-^ 0,11,9. The most stringent upper limits for presently unobserved 
decays were reported by BaBar [H, i, Q. 

DATA SET AND APPARATUS 

The study performed here includes the decays — » r/p*^°'+\ _g(o>+) — ^ r]'K*^'^'^\ 
B^ — rj'cj), B^ — > rj'uj and 5° ti't]^'^ and is based on a data sample that contains 535 xlO^ 
BB pairs, collected with the Belle detector at the KEKB asymmetric energy e"'"e" (3.5 GeV 
on 8 GeV) collider [8|. Throughout this paper, the inclusion of the charge conjugate decay 
is implied unless stated otherwise. 

KEKB operates at the T(4S') resonance (^/i = 10.58 GeV) with a peak luminosity that 
exceeds 1.7 x 10^^ cm~^s~^. The Belle detector is a large- solid- angle magnetic spectrometer 
that consists of a silicon vertex detector (SVD), a 50-layer central drift chamber (CDC), an 
array of aerogel threshold Cherenkov counters (ACC), a barrel-like arrangement of time-of- 
fiight scintillation counters (TOF), and an electromagnetic calorimeter comprised of CsI(Tl) 
crystals located inside a superconducting solenoid coil that provides a 1.5 T magnetic field. 
An iron fiux-return located outside of the coil is instrumented to detect mesons and 
to identify muons. The detector is described in detail elsewhere [9|. Two inner detector 
configurations were used. A 2.0 cm beampipe and a 3-layer SVD were used for the first 
data sample of 152 xlO^ BB pairs (Set /), while a 1.5 cm beampipe, a 4-layer SVD and a 
small-cell inner drift chamber were used to record the remaining 383 x 10^ BB pairs (Set 

EVENT SELECTION AND RECONSTRUCTION 

For what follows, unless stated otherwise, all variables are defined in the center-of-mass 
frame with the z axis anti-parallel to the positron direction. 

Charged hadrons are identified by combining information from the CDC {dE/dx), ACC 
and TOF systems. Both kaons and pions are selected with an average efficiency of 86% and 
are misidentified as pions or kaons, respectively, in 4% of the cases. 

The rj' mesons are reconstructed in the decays rj' — > rjn^Ti" (with r] 77) and f]' — * p^-y, 
except for the decays B^ rj'r], B^ rj'r]' and B^ rj'uj, which use only the t]' rjix^-K' 
channel. We define the rj' {h^*^) side as all particles involved in the decay of the rj' (/i^*^) 
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from the decay B r]'h^*\ The r], and r]' candidates on the rj' side are reconstructed 
using the mass windows given in Table [11 Mass windows used to reconstruct the h^*^ are 
given in Table [Tll In addition, we require the following: photons originating from vr" and rj 
decays are required to have energies of at least 100 MeV, photons from the rj' in rj' p°7 
have to be above 200 MeV in the laboratory frame. The transverse momenta of the vr^ for 
T]' rj^^Ti'^Ti" iji' pI+^-'j) candidates have to be greater than 100 MeV/c (200 MeV/c). 
An additional requirement on the cosine of the p^ helicity angle in i]' — s> of | cos 6h\ < 0.85 
is applied, where 6h is the angle between the momenta of one of the daughter pions of the p° 
and the rj' in the p° rest frame. The vertex of the Kg tt+tt" has to be displaced from the 
interaction point (IP) and the Kg momentum direction must be consistent with its flight 



direction as indicated in Table IIIII 11 



TABLE I: Invariant mass windows used to select intermediate states on the rj' side, a denotes a 
standard deviation of the reconstructed mass distribution. 



Mode 


Mass window 




(MeV/c2) 


in units of a 


p^ — > vr^vr^ 


[550,870] 




r/ ^ 77 


[500,570] 


+2.5/ - 3.3 


r]' — > r]TT^TT~ 


[950,965] 


±2.5 


r]' — > p^j 


[941,970] 


±2.5 



TABLE IL Invariant mass windows used to select intermediate states on the h^*^ side, a denotes 
a standard deviation of the reconstructed mass distribution. 



Mode 


Mass window 




(MeV/c2) 


in units of a 


TT^ — > 77 


[118,150] 


±2.5 


K'^ vr+vr" 


[485,510] 


±3 


p'^ — > vr+vr'^ 


[620,920] 




p'^ — > 7r~^7r~ 


[620,920] 






[820,965] 






[820,965] 






[820,965] 




(j) K+K~ 


[1010,1030] 


±3 


r/ ^ 77 


[510,575] 


±2.5 


r]' r]TT^Tr~ 


[950,965] 


±2.5 




[750,810] 


±2.5 



B meson candidates are formed by combining an t]' meson with one of the hadrons 
listed in Table HI] excluding tt^'s and K^'s. B candidates are identified using two kinematic 
variables: the energy difference, AE = Eb — -Ebeam, and the beam-energy constrained 
mass, Mbc = \J E'^^^^/c^ — {Pb/c)"^, where -Ebeam is the beam energy and Eb (Pb) is the 
reconstructed energy (momentum) of the B candidate. Signal events peak at AE = GeV 
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TABLE III: Selection criteria for the distance of cfosest approach of one of the daughter pions 
to the IP {dr) in the x-y plane, azimuthal angle between the momentum vector and the flight 
direction of the K'g candidate inferred from the production and decay vertexes {dcf), distance 
of closest approach between the two daughter tracks (z-dist.) and the flight length of the 
candidate in the x-y plane {fl). 



Momentum (GeV/c) 


dr (cm) 


d(j){vad) 


z-dist. (cm) 


fl (cm) 


< 0.5 


> 0.05 


< 0.3 


< 0.8 




0.5 - 1.5 


> 0.03 


< 0.1 


< 1.8 


> 0.08 


> 1.5 


> 0.02 


< 0.03 


< 2.4 


> 0.22 



and Mbc = Mb, where Mb is the B meson mass, with resolutions around 15 MeV and 3 MeV 
for AE and Mbc respectively. An rj mass constraint fit is applied in the rj' rj-K'^'K' subdecay 
in order to improve the AE resolution. Here the two photons from r/ — > 77 are constrained 
to have the nominal r] mass given by the Particle Data Group (PDG) [2^. Events satisfying 
the requirements Mbc > 5.22 GeV/c^ and \ AE\ < 0.25 GeV are selected for further analysis. 
After all selections are applied, depending on the decay mode between 3% and 20% of the 
events have multiple B candidates in one event. A variable is calculated to select the best 
candidate of such events. We select the B with the smallest = Xvtx + X^(^/y') 
where Xvtx estimator of the vertex quality for all charged particles not from the and 
X^iMx) = [{Mx — mx) / (Tx]"^ , where Mx (mx) is the reconstructed (nominal) mass of the 
particle candidate X (= i]' or h^*^) and ax is the standard deviation of the reconstructed X 
mass distribution as obtained from fits to MC distributions. 



BACKGROUND SUPPRESSION 

The dominant background for this analysis is continuum e^e^ ^ = d, s, c). 

Other background sources are charmless B decays such as -B — > rj'K and b ^ c decays. The 
background is 90% continuum with the remaining 10% nearly evenly split between the other 
two contributions. 

Several event shape variables are used to distinguish the spherical BB topology from 
the jet-like e~^e" qq continuum background. The thrust angle 6t is defined as the angle 
between the rj' momentum direction and the thrust axis formed by all particles not belonging 
to the reconstructed B meson. Continuum events tend to peak near | cos^^l = Ij while BB 
events have a uniform distribution. The requirement | cos 6*^1 < 0.9 is applied prior to all 
other event topology selections resulting in a signal efficiency (background reduction) of 90% 
(56%). 

Additional continuum background suppression is obtained by using modified Fox- Wolfram 



moments [12| and | cos 6*^1, where 9b is the angle between the flight direction of the recon 



structed B candidate and the beam axis. A Fisher discriminant (JF) [13[ is formed from a 



linear combination of | cos 6*^1, 5*^ [Ij] and five modified Fox- Wolfram moments. S± is the 
ratio of the scalar sum of the transverse momenta of all tracks outside a 45° cone around the 
7]' direction to the scalar sum of their total momenta. The Fisher discriminant is then com- 
bined with the B flight direction information to form an event topology likelihood function 
Cs i'^qq): where the subscript S (qq) represents signal (continuum background). The signal 
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over continuum background ratio varies over the range of the quahty parameter r of the B 
flavor tagging of the accompanying B meson. We use the standard Belle B tagging algo- 



rithm [15| , which gives the B flavor and the tagging quality r ranging from zero for no flavor 
to unity for unambiguous flavor assignment. The data is divided into three r regions and the 
likelihood ratio Tic = '^s/i'^s + '^qq) requirements are determined to maximize Ns/ ^/Nb-, 
with Ns [Nb) the expected number of signal (background), on Monte Carlo (MC) events in 
each r region separately. More stringent selections are imposed for the first r region at zero 
while looser criteria are used for r close to one. More stringent selections are applied for de- 
cays with large continuum contribution such as -B ^ rj'p, while relatively clean decays such 
as B^ rj'cf) have very loose requirements. The signal efficiencies (continuum background 
reduction) lie in the range of 42%-88% (98%-45%). 

Contributions from other charmless B decays can contaminate the signal when a pion is 
misidentified as a kaon or when a random pion is added or missed. The dominant contribu- 
tion for such misidentified events originates from B — rj'K decays. For the decays B rj'p, 
B rj'K* and B^ — > rj'uj the B — *• r]'K contamination is significant. For these decays we 
construct an alternative B meson hypothesis assuming that it originates from a i? — > rj'K 
decay. We then veto an event if the alternative AE variable is within a decay-dependent 
window around AE = GeV and Mbc > 5.27 GeV/c. The selection is optimized for each 
decay and results in negligible signal suppression (< 0.5%) while removing around 80% of 
the B — > rj'K background. 



MEASUREMENT OF BRANCHING FRACTIONS 

The branching fractions are obtained using an extended unbinned maximum-likelihood 
fit to the AE and Mbc distributions of selected events. This fit is performed simultaneously 
in the i]' — > r/vr+vr" and rj' — >■ p°7 subdecay channels for all B decay modes, where applicable. 
In the case of B^ — > rj'K*~^ the two K*^ subdecay modes (thus four subdecay channels in 
total) are fitted simultaneously. The extended likelihood function used is: 



l{Ns,nb,) = — - — n 

■ i=l 



NsPs{AE,,M^,^) 
+ T.^B,PB,{m,M^,j], (1) 



where A^^ (Nbj) is the number of signal events (background events of source j) with prob- 
ability density functions (PDFs) Ps {Psj)- The index i runs from 1 to the total number of 
events TV in the selected sample. 

The branching fraction B is determined by maximizing the combined likelihood for both 
data sets and all subdecays with B constrained to be the same for the subdecays. The 
number of signal events (Ns) for each decay mode is calculated by A^^ = B [Nbb{I) ^t{I) + 
Nbb{II) ^t{II)], where NBsiM) is the number of BB produced for set /c = J or II and et(/c) 
is the total reconstruction efficiency including subdecay branching fractions for set k. 

The reconstruction efficiencies are determined from signal MC samples using, the EvtGen 
package 1^ with final state radiation simulated by the PHOTOS package [l7| (thus mea- 



suring B — > r]'h^*\'^)). The efficiencies are calculated separately for Set I and Set //. The 
absolute efficiency for Set // is typically about 0.5% larger than for Set / (for efficiencies 
averaged over the two sets see Tables HVl and IVT) . Corrections due to differences between 
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data and MC are included for the charged track identification and photon, vr^ and rj recon- 
structions, resulting in an overall correction factor between 0.88 and 0.99 depending on the 
decay mode. We assume the numbers of B^B~ and B^B^ pairs to be equal in the original 
data sample. 

In the fit to the data we consider a signal component and three types of background 
components: continuum events, events from other B meson decays via the dominant b ^ c 
transition and from charmless B decays. 

For both signal and continuum background AE and Mbc are uncorrelated and we use two 
independent functions to describe the shapes of AE and Mbc- To model the signal, we use a 
Gaussian with an exponential tail, the so-called Crystal Ball-line (CBline) function 1^, plus 
a Gaussian in AE, while Mbc is described by a single CBline function. The shape parameters 
are fixed from the signal MC study. Corrections for MC-data discrepancies determined from 
control samples of B~^ rj'K^ and B^ D^n^, where K~7i^ and —>■ K~7T~^7r^, 

are applied to the mean and width of the CBline functions. 

Continuum background is modelled by a first-order polynomial for AE and an ARGUS 
function [l^ for Mbc. The continuum shape parameters, that are allowed to fioat in all 
modes, are the slopes of the polynomial and ARGUS function. The shapes for charmless 
B decays remaining after applying the vetoes and b ^ c backgrounds are modelled by two- 
dimensional smoothed histograms. The sizes of background contributions other than the 
continuum background are fixed to the values expected from MC studies. 

The resulting AE and Mbc projections are shown in Figs. [1] and [2l The reconstruction 
efficiencies and fit results are given in Tables [IV] and IVl 



TABLE IV: Average efficiencies (e) for the two data sets for t]' t/vt+tt- and t]' p^j, total 
efficiencies (et) with systematic errors of secondary branching fractions included, signal yield (Ns) 
with statistical errors only and the 90% confidence level upper limit on the branching fraction in 
units of 10~6 including systematic errors for each decay of this analysis (UL) and latest results 
from BaBar in units of 10-^. 





5° ^ 


B+ i]'p+ 


^0 ^ ^/^^*0 


B+ 7]'K*+ 


B^ 


e{7]7nr) [%] 


7.0 ±0.1 


5.9 ±0.1 


8.5 ±0.1 


4.5 ±0.1 


12.9 ±0.1 


<Pl) [%] 


5.4 ±0.1 


3.9 ±0.1 


5.9 ±0.1 


2.2 ±0.1 


7.4 ±0.1 


et(??^vr) [%] 


1.13 ±0.02 


0.93 ±0.02 


0.92 ±0.01 


0.35 ±0.01 


1.08 ±0.01 


et(P7) [%] 
Ns 

UL [10-6] 


1.51 ±0.03 
< 1.3 


1.07 ±0.02 

1 R c+23.3 
< 5.8 


1.09 ±0.02 
14 2+^-1 
< 2.6 


0.30 ±0.01 
_fi 4+10-9 

< 2.9 


1.08 ±0.02 
< 0.5 


BaBar[10-6] 


< 3.7 


< 14 


3.8 ±1.1 ±0.5 


4:.9t\j ±0.8 


< 4.5 



SYSTEMATICS 

Systematic errors on the branching fractions are estimated with various high statistics 
data samples. The dominant sources are the uncertainties in the reconstruction efficiency 
of charged tracks (3-4%), the uncertainties in the reconstruction efficiencies of r] mesons, 
7r°'s and photons (3-6%) and the Kg reconstruction efficiency uncertainty (4%). Other 
systematic uncertainties arise from signal MC statistics (2%), likelihood ratio selections 
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TABLE V: Average efficiencies (e) for the two data sets for ?/ r/Tr+vr", total efficiencies (et) 
with systematic errors of secondary branching fractions included, signal yield (Ns) with statistical 
errors only and the 90% confidence level upper limit on the branching fraction in units of 10~^ 
including systematic errors for each decay of this analysis (UL) and latest results from BaBar in 

units of 10"^. 

B° 7]'ri B° -q'ri' S° -q'uj 

e(?77r7r) [%] 5.7 ±0.1 4.8 ± 0.1 7.5 ±0.1 

et(??7r7r) [%] 0.37 ± 0.007 0.16 ±0.003 1.09 ± 0.02 
Ns l.Otii -6.3i:? 0.91^^ 

UL [10-6] < 4.5 < 6.5 < 2.2 

BaBar[10"6] < 1.7 < 10 < 2.8 



± data 



combined 



signal 



udsc 



BB 



- rare B 



> 

0) 




2 -0.1 



0.1 0.2 




AE [GeV] 



AE [GeV] 



B 



AE [GeV] 
^0 ^ ^/^*0 



AE [GeV] 

5+ ^ TiK' 



LU 



150 



1001 




5.22 5.24 5.26 5.28 5.3 5.22 5.24 5.26 5.28 5.3 5.22 5.24 5.26 5.28 5.3 5.22 5.24 5.26 5.28 5.3 

Mbc [GeV/c] A/bc [GeV/c] M^^ [GeV/c] M^^ [GeV/c] 



FIG, 
5°- 



1: AE (upper) and Mbc (lower) distributions for (from left to right) B^ — > r( ff' ^ B^ 



7]'K*^ and B+ ij'K* 



(2%), uncertainties of the subdecay branching fractions as given by the PDG (1.7-3.0%), the 
number of BB mesons produced (1.4%) and the uncertainty from particle identification (0.5- 
1.3%). In addition, we calculate systematic uncertainties for the fitting procedure by varying 
all PDF shape parameters by ±lcr. Background normalization systematic uncertainties are 
estimated by varying the background normalizations by 20%-50% while those for AE/M^^c 
corrections are obtained by varying the corrections by one standard deviation. Since for most 
decays the fits yield branching fractions close to zero, we use absolute errors in these cases. 
Fractional errors are translated into absolute values by multiplying the obtained upper limit 
value by the fractional error. The combined absolute errors are decay dependent and lie in 
the range (0.01 - 4.93 )xlO~^. The total systematic uncertainties are listed in Table IVll 
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+ data 



combined 



— signal 



udsc 



BB 



— ■ rare B 





AE [GeV] 



B 







1 



AE [GeV] 
50 



0.1 0.2 -0.2 -0.1 0.1 0.2 -0.2 -0.1 0.1 0.2 

AE [GeV] AE [GeV] 

50 _^ I I 



r]'r] 



r]'uj 



20 



10 




5.22 5.24 5.26 5.28 5.3 5.22 5.24 5.26 5.28 5.3 5.22 5.24 5.26 5.28 5.3 5.22 5.24 5.26 5.28 5.3 



Mbc [GeV/c] 



A/bc [GeV/c] 



Mbc [GeV/c] 



[GeV/c] 



FIG. 



2: AE (upper) and (lower) distributions for (from left to right) B^ if(j), B^ 
> rj'rj' and B^ 



v'v, 



r]'uj. 



TABLE VI: Total systematic uncertainties for each decay. Listed are combined errors for fitting, 
efficiency related errors and the error in the number of BB events. Conservatively, we take the 
total systematic error to be the linear sum of these. All errors are in absolute values in units of 
10-7. ^^^^^^^^^^^^^^^^^^^^^^^ 

Decay Fitting Efficiency i^BB Total 

o~7Z7j> +0.33 n 07 n 09 +0.42 

-1.76 ^-^^ -1.85 



Efficiency 


#BB 


0.07 


0.02 


0.32 


0.06 


0.16 


0.04 


0.21 


0.04 


0.03 


0.01 


0.26 


0.05 


0.29 


0.05 


0.16 


0.03 



+ 0.32 0.06 

. ^' 1^*0 +0.04 p) . +0.24 

-0.03 U-J^O U.U4 _Q23 

+0.84 r, 21 04 

-10.10 ^■'^^ ^-^^ -10.35 



5^ ^ 

r]'p 

B° r]'K 
B+ r]'K 

B° r^4) ±0.10 0.03 0.01 ±0.14 

5° rlr] 
B° r(ri 
B° r]'uj 



?0 _ „i„ +2.43 r. Of. p Qc: +2.74 

-0.36 U-^O ^-^^ -0.67 



^ +24.85 n 9Q n nf; +25.19 

-48.94 U-^y "-'•"-'^ -49.28 

° i^i^ 0.16 0.03 tl-ll 



UPPER LIMIT CALCULATION 



Since no decay has more than 2a significance [21[, we calculate upper limits on the 
branching fractions by integrating the likelihood function starting at i3 = using a Bayesian 
approach assuming a uniform distribution for B > 0. We set the upper limit when the 
integral reaches 90% of the total area under the likelihood function. The systematic error 
is accounted for by folding the systematic error into the width of the likelihood distribution 
(Eq. [1]) when integrating the likelihood. Thus the upper limit (UL) is calculated with the 
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r]'(p J^" ^ Tj'T] J^" ^ Tj'T]' J^" Tj'uj 

FIG. 3: Distributions of likelihood vs. branching fraction for each decay. The systematic error is 
included as described in the text. Two dashed lines indicate B = and the 90% confidence level 
upper limit. 



formula: 



0.9, 



(2) 



where Lsys{Ns, Nb^) is the likelihood function with its width increased by the systematic 
error. The likelihood distribution is shown in Fig. [3] for each decay mode. 



The thus calculated upper limits are 0.5 x 10 ^ for 



7] < 



1.3 X lO"'' for 



/ ^0 



VP 



and in the range 2.2-6.5 x 10~^ for other modes, as given in Tables IIVI and |Vl We note 
that our upper limits for B^ — > 7]'K*^ and B^ — >■ r]'K*^ are below the central values of the 
BaBar measurement. 



SUMMARY 

In summary, no signal was observed with more than 2a significance and stringent upper 
limits in the range (0.5 - 6.5) x 10~® for the decays B — > rj' B —>■ rj'K*, B^ — >■ rj'cj), 
B^ — *• ?7'?7'^')and B'^ rj'uj have been given. All limits except B^ rj'r] are the most stringent 
upper limits presently available. Our upper limits for B — > rj'K* are below BaBar's central 
value. 
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